Encoding apparatus and method, image capture apparatus, and storage medium

ABSTRACT

An apparatus comprises a generating unit configured to generate a plurality of pieces of RAW data for respective exposure times from RAW data obtained from a sensor that can perform shooting at an exposure time that is different for each pixel, and an encoding unit configured to encode the generated plurality of pieces of RAW data.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The aspect of the embodiments relates to a technique for encoding andrecording an image obtained by an image sensor that can control theexposure time for each pixel.

Description of the Related Art

In known image capture apparatuses, raw image information (RAW data)obtained by capturing performed by an image sensor is converted tosignals constituted by luminance and color differences by applyingdebayering processing (demosaicing processing), and so-called developingprocessing such as noise removal, optical distortion correction, andimage optimization is performed on each signal. Also, in general, aluminance signal and color difference signals that have been subjectedto the developing processing are compression-encoded and recorded in arecording medium.

On the other hand, there are also image capture apparatuses that storeimage capture data (RAW data) that is in a state of immediately afterbeing output from the image sensor and has not been subjected todeveloping processing in a recording medium. When the RAW data isrecorded, data saving can be performed in a state of retaining abundantnumber of color tones without the color information from an image sensorbeing degraded, and therefore editing with a high degree of freedom canbe performed. However, there may be issue in that the recording dataamount of RAW data is huge, and a lot of free space is needed in therecording media. Therefore, it may be desirable that the RAW data isalso subjected to compression encoding, and is recorded while the dataamount being suppressed.

Incidentally, an image capture device is known with which, as a resultof arranging pixels that are different in exposure time on the sameplane, as disclosed in Japanese Patent Laid-Open No. 2013-21660, animage having a wide dynamic range can be obtained with one instance ofshooting, as a device for obtaining a high dynamic range image. Asynthesizing method for generating a high dynamic range image at thetime of development when such an image capture device is used isdisclosed in Japanese Patent Laid-Open No. 2013-21660.

However, in the known technique disclosed in Japanese Patent Laid-OpenNo. 2013-21660 described above, a method of encoding RAW data beforesubjected to synthesizing is not disclosed.

Also, when an image capture device as described in Japanese PatentLaid-Open No. 2013-21660 is used, if RAW data before subjected tosynthesizing is tried to be encoded, because the level differencebetween pixels that are different in exposure time and are arranged onthe same plane is large, a large amount of high frequency components aregenerated, and as a result, the coding efficiency drops.

SUMMARY OF THE DISCLOSURE

According to a first aspect of the embodiments, there is provided anapparatus comprising: at least one processor; and a memory coupled tothe at least one processor, the memory having instructions that, whenexecuted by the at least processor, perform operations as: a generatingunit configured to generate a plurality of pieces of RAW data forrespective exposure times from RAW data obtained from a sensor that canperform shooting at an exposure time that is different for each pixel;and an encoding unit configured to encode the generated plurality ofpieces of RAW data.

According to a second aspect of the embodiments, there is provided anapparatus comprising: a sensor that can control the exposure time foreach pixel: and an encoding apparatus comprising: at least oneprocessor; and a memory coupled to the at least one processor, thememory having instructions that, when executed by the at leastprocessor, perform operations as: a generating unit configured togenerate a plurality of pieces of RAW data for respective exposure timesfrom RAW data obtained from an image sensor that can perform shooting atan exposure time that is different for each pixel; and an encoding unitconfigured to encode the generated plurality of pieces of RAW data.

According to third aspect of the embodiments, there is provided a methodcomprising: generating a plurality of pieces of RAW data for respectiveexposure times from RAW data obtained from a sensor that can performshooting at an exposure time that is different for each pixel; andencoding the generated plurality of pieces of RAW data.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of adigital camera that is a first embodiment of an encoding apparatus ofthe disclosure.

FIG. 2 is a diagram illustrating a pixel array of an image capture unit.

FIG. 3 is a diagram illustrating the pixel array of the image captureunit and setting of exposure time.

FIG. 4A is a diagram illustrating a separating method of RAW data in thefirst embodiment.

FIG. 4B is a diagram illustrating the separating method of RAW data inthe first embodiment.

FIG. 4C is a diagram illustrating the separating method of RAW data inthe first embodiment.

FIG. 4D is a diagram illustrating the separating method of RAW data inthe first embodiment.

FIG. 5 is a diagram illustrating RAW data output when the exposure timesof pixels are the same.

FIG. 6 is a block diagram illustrating a configuration of a RAW encodingunit.

FIGS. 7A and 7B are diagrams illustrating an example of frequencytransform (subband division).

FIG. 8 is a diagram illustrating an example of a unit for generating aquantization parameter.

FIG. 9A is a diagram illustrating an exemplary generation of thequantization parameter.

FIG. 9B is a diagram illustrating an exemplary generation of thequantization parameter.

FIG. 9C is a diagram illustrating an exemplary generation of thequantization parameter.

FIGS. 10A and 10B are diagrams illustrating a separating method of RAWdata in a second embodiment.

FIG. 11 is a diagram illustrating a separating method of RAW data in athird embodiment.

FIG. 12 is a diagram illustrating a pixel array and a setting ofexposure time in a fourth embodiment

FIG. 13 is a diagram illustrating rearrangement of a pixel array in afourth embodiment.

FIG. 14 is a block diagram illustrating a configuration of a RAWencoding unit in a fifth embodiment.

FIGS. 15A and 15B are diagrams illustrating frequency transform (subbanddivision).

FIG. 16 is a processing block diagram for describing HDR synthesizingprocessing.

FIGS. 17A to 17C are diagrams illustrating a synthesizing ratio in theHDR synthesizing processing when a long exposure image is of correctexposure.

FIGS. 18A to 18C are diagrams illustrating a synthesizing ratio in theHDR synthesizing processing when a short exposure image is of correctexposure.

FIGS. 19A to 19C are diagrams illustrating an exemplary setting of thequantization parameter.

FIGS. 20A to 20C are flowcharts illustrating a quantization processingprocedure of the fifth embodiment.

FIGS. 21A to 21C are flowcharts illustrating a quantization processingprocedure of a sixth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the disclosure. Multiple features are described inthe embodiments, but limitation is not made such that requires all suchfeatures, and multiple such features may be combined as appropriate.Furthermore, in the attached drawings, the same reference numerals aregiven to the same or similar configurations, and redundant descriptionthereof is omitted.

First Embodiment

FIG. 1 is a block diagram illustrating a functional configuration of adigital camera 100 that is a first embodiment of an encoding apparatusof the disclosure. The digital camera 100 includes an image capture unit101, a separating unit 102, a RAW encoding unit 103, a recordingprocessing unit 104, a recording medium 105, a memory I/F (memoryinterface) 106, and a memory 107.

The image capture unit 101 includes a lens optical system that includesan optical lens, an aperture, a focus controller, and a lens drivingunit, and is capable of optical zoom, and an image sensor in which aplurality of pixels each including a photoelectric conversion elementare two-dimensionally arranged.

The image sensor performs photoelectric conversion, in each pixel, on asubject image formed by the lens optical system, and also performsanalog/digital conversion with an A/D conversion circuit, and outputs adigital signal (pixel data, RAW data) in units of pixel. A CCD imagesensor, a CMOS image sensor, or the like is used as the image sensor.

Note that, in the present embodiment, each pixel of the image sensor isprovided with one of R (red), G1/G2 (green), and B (blue) color filters,as shown in FIG. 2. Note that the RAW data output from the image captureunit 101 is stored in the memory 107 via the memory I/F 106.

The separating unit 102 is a circuit or a module for separating the RAWdata obtained by the image capture unit 101 into pieces of RAW data forrespective exposure times. The RAW data stored in the memory 107 is readout via the memory I/F 106, and is separated into pieces of RAW data forrespective exposure times, which are output to the RAW encoding unit103.

The RAW encoding unit 103 is a circuit or a module that performscomputational operations on RAW data, and encodes the RAW data inputfrom the separating unit 102. The RAW encoding unit 103 stores codeddata generated by encoding in the memory 107 via the memory I/F 106.

The recording processing unit 104 reads out various types of data suchas coded data that are stored in the memory 107 via the memory I/F 106,and records the read-out data in the recording medium 105. The recordingmedium 105 is a recording media that is constituted by a large capacityrandom access memory such as a nonvolatile memory.

The memory I/F 106 mediates memory access requests from the processingunits, and performs reading/writing control with respect to the memory107. The memory 107 is a volatile memory such as an SDRAM, and functionsas storage means. The memory 107 provides a storage area for storing theaforementioned various types of data such as image data and sound data,or various types of data output from the processing units thatconstitute the digital camera 100.

Next, the pixel arrangement structure of the image capture unit 101 willbe described with reference to FIG. 2. As shown in FIG. 2, the imagecapture unit 101 is characterized in that R pixels, G1 pixels, G2pixels, and B pixels are arranged in units of 2×2 pixels, and the samecolor is arranged in each 2×2 pixels. The image capture unit 101 has astructure in which the total 4×4 pixels is a minimum unit, and theminimum unit repeatedly arranged.

The setting of exposure time in an image sensor that has the pixelarrangement structure shown in FIG. 2, and in which the exposure timecan be controlled for each pixel (shooting is possible with an exposuretime that is different for each pixel) will be described with referenceto FIG. 3. As shown in FIG. 3, the horizontal direction is denoted by x,the vertical direction is denoted by y, the column number is representedby an x coordinate, and the row number is represented by a y coordinate.The numbers with parentheses indicate the coordinates indicating theposition of each pixel on the image sensor. Also, white pixels representshort exposure pixels, and gray pixels represent long exposure pixels.In the present embodiment, short exposure pixels that perform shortexposure and long exposure pixels that perform long exposure are set ina zigzag manner in the column direction, as shown in FIG. 3.

For example, the setting of exposure time regarding four R pixels at anupper left end in FIG. 3 is as follows. R(1,1) is a short exposurepixel, R(2,1) is a long exposure pixel, R(1,2) is a long exposure pixel,and R(2,2) is a short exposure pixel. In this way, a short exposurepixel and a long exposure pixel are alternatingly set in each column,and a short exposure pixel and a long exposure pixel are alternatinglyset in each row. When the short exposure pixels are followed in the ydirection, in the first and second columns, in the first row from theabove, the first column is a short exposure pixel, in the second row,the second column is a short exposure pixel, in the third row, the firstcolumn is a short exposure pixel, and in the fourth row, the secondcolumn is a short exposure pixel. Similarly, when the long exposurepixels are followed in the y direction, in the first and second columns,in the first row from the above, the second column is a long exposurepixel, in the second row, the first column is a long exposure pixel, inthe third row, the second column is a long exposure pixel, and in thefourth row, the first column is a long exposure pixel.

As described above, the pixel arrangement structure and the setting ofthe exposure time are set such that pixels of the same color are set inunits of 2×2 pixels, and two short exposure pixels (one of two exposuretimes) and two long exposure pixels (the other of two exposure times)are arranged in those 4 pixels.

Here, if encoding is tried to be performed in a state in which the RAWdata is as obtained by the image capture unit 101, that is, in a statein which pixels that are different in exposure time are mixed, becausethe level difference between pixels that are different in exposure timeis large, a large amount of high frequency component is generated, and arecording data amount of the RAW data increases. Therefore, in thepresent embodiment, the RAW data is separated into pieces of RAW data ofthe respective exposure times by the separating unit 102, and generationof high frequency component is suppressed by matching the level betweenpixels, and with this, the recording data amount of RAW data is reduced.

Next, the separating method will be described with reference to FIGS. 4Ato 4D. The separating unit 102 separates the RAW data input from theimage capture unit 101 into Bayer arrangement structure RAW data that isconstituted by short exposure pixels and Bayer arrangement structure RAWdata that is constituted by long exposure pixels, as shown in FIGS. 4Ato 4D, and outputs the separated two pieces of RAW data to the RAWencoding unit 103.

Specifically, the RAW data constituted by short exposure pixels isseparated into two planes of short exposure RAW data that areillustrated by RAW data 401 a in FIG. 4A and RAW data 401 b in FIG. 4B.The RAW data 401 a is short exposure RAW data that is configured byextracting short exposure pixels each marked by a rhombus inodd-numbered rows and in odd-numbered columns, as shown in FIG. 4A.Also, the RAW data 401 b is short exposure RAW data that is configuredby extracting short exposure pixels each marked by a rhombus ineven-numbered rows and in even-numbered columns, as shown in FIG. 4B.

Similarly, the RAW data constituted by long exposure pixels is separatedinto two planes of long exposure RAW data that are illustrated by RAWdata 401 c in FIG. 4C and RAW data 401 d in FIG. 4D. The RAW data 401 cis long exposure RAW data that is configured by extracting long exposurepixels each marked by a rhombus in odd-numbered rows and ineven-numbered columns, as shown in FIG. 4C. Also, the RAW data 401 d islong exposure RAW data that is configured by extracting long exposurepixels each marked by a rhombus in even-numbered rows and inodd-numbered columns, as shown in FIG. 4D. The RAW encoding unit 103separately encodes RAW data 401 a, 401 b, 401 c, and 401 d that areinput in a Bayer arrangement manner from the separating unit 102.

Note that the separating method of the separating unit 102 when theexposure time is different between pixels arranged on the same plane hasbeen described above using the pixel array in FIG. 2. Next, theprocessing to be performed by the separating unit 102 when the exposuretimes of the pixels are all the same will be described with reference toFIG. 5.

In this case, the separating unit 102 configures RAW data 501 bycalculating a pixel average value of each four pixels of the same colorcomponent that are marked by a gray rhombus, as shown in FIG. 5, withrespect to the RAW data obtained by the image capture unit 101, andoutputs the RAW data 501 to the RAW encoding unit 103. Specifically, asshown in the following Formulas 1 to 4, separation is performed bycalculating addition averages for each color component.

{R(1,1)+R(2,1)+R(1,2)+R(2,2)}/4= R (1,1)  Formula 1

{G1(3,1)+G1(4,1)+G1(3,2)+G1(4,2)}/4= G1(2,1)  Formula 2

{G2(1,3)+G2(2,3)+G2(1,4)+G2(2,4)}/4= G2(1,2)  Formula 3

{B(3,3)+B(4,3)+B(3,4)+B(4,4)}/4= B (2,2)  Formula 4

Next, the detailed configuration of the RAW encoding unit 103 thatperforms processing on the short exposure RAW data 401 a and 401 b andthe long exposure RAW data 401 c and 401 d and the processing flow willbe described with reference to the block diagram shown in FIG. 6.

The RAW encoding unit 103 includes a channel transform unit 601, afrequency transform unit 602, a quantization parameter generating unit603, a quantization unit 604, an encoding unit 605, and a quantizationparameter encoding unit 606.

The channel transform unit 601 transforms RAW data configured as in theBayer arrangement that is input from the separating unit 102 into aplurality of channels. For example, transformation is performed intofour channels separately for R, G1, G2, and B of the Bayer arrangement.Alternatively, with respect to R, G1, G2, and B, transformation intofour channels is performed by further performing calculation using thefollowing transform formulas 5 to 8.

Y=(R+G1+G2+B)/4  Formula 5

C0=R−B  Formula 6

C1=(G0+G1)/2=(R+B)/2  Formula 7

C2=G0−G1 Formula 8

Note that, an exemplary configuration for transforming into fourchannels is shown here, but the number of channels and the transformmethod are not limited thereto.

The frequency transform unit 602 performs frequency transform processingby discrete wavelet transform at a predetermined resolution level(hereinafter, denoted as “lev”) for each channel, and outputs generatedsubband data (transform coefficient) to the quantization parametergenerating unit 603 and the quantization unit 604.

FIG. 7A shows a filter bank configuration for realizing the discretewavelet transform regarding the subband division processing at lev=1.When the discrete wavelet transform processing is executed in thehorizontal and vertical directions, division into one low frequencysubband (LL) and three high frequency subbands (HL, LH, HH) isperformed, as shown in FIG. 7B.

The transfer functions of the low pass filter (hereinafter, denoted as“lpf”) and the high pass filter (hereinafter, denoted as “hpf”) shown inFIG. 7A are respectively shown in Formulas 9 and 10.

lpf(z)=(−z ⁻²+2z ⁻¹+6+2z ¹ −z ²)/8   Formula 9

hpf(z)=(−z ⁻¹+2−z ¹)/2  Formula 10

When lev is larger than 1, subband division is hierarchically executedwith respect to the low frequency subband (LL). Note that, here, thediscrete wavelet transform is configured by a five tap lpf and a threetap hpf, as shown in Formulas 9 and 10, but there is no limitationthereto, and a filter configuration in which the number of taps and thecoefficients are different may be adopted.

The quantization parameter generating unit 603 generates a quantizationparameter for performing quantization processing on the subband data(transform coefficient) generated by the frequency transform unit 602for each certain predetermined subband data unit. The generatedquantization parameter is input to the quantization parameter encodingunit 606, and is also supplied to the quantization unit 604.

The quantization unit 604 performs quantization processing on thesubband data (transform coefficient) output from the frequency transformunit 602 based on the quantization parameter supplied from thequantization parameter generating unit 603, and outputs the quantizedsubband data (transform coefficient) to the encoding unit 605.

The encoding unit 605 performs predictive difference entropy coding ofthe quantized subband data (transform coefficient) output from thequantization unit 604 for each subband in a raster scan order, andstores the generated encoded RAW data to the memory 107. Note that othermethods may be used as the prediction method and the entropy codingmethod.

The quantization parameter encoding unit 606 is a processing unit forperforming encoding on the quantization parameter input from thequantization parameter generating unit 603. The quantization parameteris encoded using the encoding method in common with the encoding unit605, and the generated encoded quantization parameter is stored to thememory 107.

Next, the relationship between the subband data, the channel data, andthe RAW data will be described with reference to FIG. 8 when thequantization parameter is generated assuming that the aforementionedpredetermined subband unit is 4×4.

The 4×4 subband corresponds to 8×8 pixels for each channel, as shown inFIG. 8, and also corresponds to a block corresponding to 16×16 pixels ofeach RAW data. Therefore, in this case, the quantization parameter is tobe stored in the memory 107 for each RAW data block corresponding to16×16 pixels, in the short exposure RAW data 401 a and 401 b and thelong exposure RAW data 401 c and 401 d.

Note that it is effective to apply the same quantization parameter tothe short exposure RAW data 401 a and 401 b, and to the long exposureRAW data 401 c and 401 d in order to reduce the data amount of thequantization parameter. In this case, the data amount can be reduced tohalf. Also, in the present embodiment, using the quantization parameterthat is generated at an exposure time closer to the correct exposure asa reference, and the other quantization parameter is calculated, inorder to further reduce the data amount. With this, the data amount ofthe quantization parameter can be reduced to quarter. Here, the reasonwhy the quantization parameter that is generated at an exposure timecloser to the correct exposure is used as the reference is because, withan overexposure or underexposure image in which blown out highlights orblacked out occurs, the quantization parameter cannot be generatedaccording to an accurate feature of the subject.

When short exposure is closer to the correct exposure, as a specificexample, the calculation formula for calculating the quantizationparameter for the long exposure RAW data with the quantization parametergenerated regarding the short exposure RAW data being the reference isshown in Formula 11.

L_Qp=α×S_Qp+β  Formula 11

-   Here,-   L_Qp: quantization parameter for long exposure RAW data-   S_Qp: quantization parameter for short exposure RAW data-   α: slope-   β: intercept.

Note that, in the present embodiment, the quantization parameter for thelong exposure RAW data is calculated with the quantization parametergenerated for the short exposure RAW data being the reference. However,the quantization parameter for the short exposure RAW data may becalculated with the quantization parameter generated for the longexposure RAW data being the reference. Also, the quantization parametermay be calculated by setting α and β for each of short exposure and longexposure while using neither of short exposure and long exposure as thereference.

Next, the determination method of α and β shown in Formula 11 will bedescribed. Although α and β may be any values, in the presentembodiment, a detailed parameter determination method will be described.When the short exposure is assumed to be closer to the correct exposure,as in the example described above, in the long exposure, overexposure isachieved because the exposure time is longer than the short exposure.Therefore, it is highly possible that, regarding an area in whichbrightness is medium to bright at the short exposure, the pixel valuereaches a saturation level at the long exposure, and the pixel valueaccording to the brightness of a subject cannot be output. On the otherhand, regarding a dark area, it is possible to obtain detailedinformation relative to the short exposure. Therefore, the quantizationparameter for the long exposure RAW data is increased relative to theshort exposure regarding an area that is determined to be an area inwhich the brightness is medium to bright in the short exposure RAW data.Also, the same parameter is set regarding an area that is determined tobe dark, and as a result, the data amount of the quantization parametercan be reduced while ensuring the image quality.

Specific description will be given with reference to FIGS. 9A to 9C.FIG. 9A shows one exemplary setting of the quantization parameteraccording to the brightness of the short exposure RAW data, in the shortexposure RAW data. Also, FIG. 9B shows exemplary setting of thequantization parameter according to the brightness of the short exposureRAW data, in the long exposure RAW data. Note that the brightness indexmay be evaluated using a ILL subband corresponding to the quantizationparameter generation unit described above. The magnitude relationshipbetween the quantization parameters is shown in Formulas 12 to 14.

Q0<Q1<Q2  Formula 12

Q1<Q3   Formula 13

Q2<Q4   Formula 14

First, the quantization parameters in the short exposure RAW data areset such that the quantization parameter decreases as the darknessincreases, considering the visual property (Q0<Q1<Q2). In contrast, inthe long exposure RAW data, the quantization parameters are set suchthat Q0 is set in an area corresponding to a dark portion in the shortexposure RAW data so as to be the same as in the short exposure RAWdata, and the quantization parameters are set in an area correspondingto medium to bright portions so as to be increased relative to the shortexposure RAW data (Q1<Q3, Q2<Q4).

FIG. 9C shows a graph for calculating the quantization parameters forthe long exposure RAW data with the quantization parameters generatedfor the short exposure RAW data being the reference. The horizontal axisshows the quantization parameter (S_Qp) for the short exposure RAW data,and the vertical axis shows the quantization parameter (L_Qp) for thelong exposure RAW data. α and β shown in Formula 11 may be set so as toachieve the relationship of Formulas 12 to 14.

Note that α and β are stored in the memory 107 similarly to the codeddata, and is recorded in the recording medium 105 via the memory I/F 106along with the coded data. Also, a flag indicating which of the shortexposure and long exposure is of the quantization parameter to be thereference is stored in the memory 107 and is recorded in the recordingmedium 105 via the memory I/F 106 along with the coded data. Note thatthe flag may not be included when α and β are set for each exposure timewhile using neither of the short exposure and the long exposure as thereference.

Also, when any of the cases described above is handled, a configurationmay be such that a flag indicating whether or not an exposure time to bea reference is included, and next, if the exposure time to be areference is present, a flag indicating which of short exposure and longexposure is the reference is included. In this case as well, each flaginformation is stored in the memory 107, and is recorded in therecording medium 105 via the memory I/F 106 along with the coded data.

As described above, in the present embodiment, the separating unit 102separates RAW data into pieces of data of respective exposure times, thelevel difference between pixels that are to be encoded is eliminated,and with this, a high frequency component is suppressed, and as aresult, the recording data amount of RAW data can be reduced. Also,using a quantization parameter calculated for one RAW data as areferent, the quantization parameter for the other RAW data of thedifferent exposure time is determined, and as a result, the recordingdata amount of RAW data can be reduced.

Second Embodiment

Next, a second embodiment of the disclosure will be described. In thesecond embodiment, the separating method of RAW data in a separatingunit 102 is different from that of the first embodiment. Note that theconfiguration of a digital camera of the second embodiment is the sameas that of the first embodiment, and therefore redundant descriptionwill be omitted, and the difference will be described.

In the first embodiment, the pieces of RAW data obtained by separating,in the separating unit 102, the pixels into groups of pixels of the sameexposure time, that is, specifically, into two planes of RAW dataconstituted by short exposure pixels and two planes of RAW dataconstituted by long exposure pixels, are output to the RAW encoding unit103.

In contrast, in the second embodiment, a method will be described inwhich, in order to further reduce the data amount, in the separatingunit 102, pixel values of pixels of the same exposure time and of thesame color component that are present in the vicinity are added, and anaverage pixel value is calculated and output to a RAW encoding unit 103.

The processing of the separating unit 102 in the present embodiment willbe described with reference to FIGS. 10A and 10B. With respect to RAWdata that is input from an image capture unit 101 and in which pixels ofdifferent exposure times are mixed, the separating unit 102 calculatesan addition average of pixel values of pixels that are enclosed by eachrectangle shown in FIG. 10A, that is, pixels that are short exposurepixels and of the same color component, and separates into shortexposure RAW data 1001 a. Specifically, as shown in the followingFormulas 15 to 18, separation is made by calculating addition averagesfor each color component.

{R(1.1)+R(2,2)}/2= R′ (1,1)  Formula 15

{G1(3.1)+G1(4,2)}/2= G1′(2,1)  Formula 16

{G2(1.3)+G2(2,4)}/2= G2′(1,2)  Formula 17

{B(3.3)+B(4,4)}/2= B′ (2,2)  Formula 18

Similarly, separation into long exposure RAW data 1001 b is made bycalculating an addition average of pixels that are enclosed by eachrectangle shown in FIG. 10B, and are long exposure pixels and of thesame color component. Specifically, as shown in the following Formulas19 to 22, separation is made by calculating addition averages for eachcolor component.

{R(2.1)+R(1,2)}/2= R′ (1,1)  Formula 19

{G1(4.1)+G1(3,2)}/2= G1′(2,1)  Formula 20

{G2(2.3)+G2(1,4)}/2= G2′(1,2)  Formula 21

{B(4.3)+B(3,4)}/2= B′ (2,2)  Formula 22

As described above, in the second embodiment, the RAW data obtained bythe image capture unit 101 is separated by calculating addition averagesin the separating unit 102, and as a result, the data amount to beoutput to the RAW encoding unit 103 can be reduced to half relative tothe first embodiment.

Third Embodiment

Next, a third embodiment of the disclosure will be described. In thethird embodiment, the separating method of RAW data in a separating unit102 is different from those of the first and second embodiments. Notethat the configuration of a digital camera of the present embodiment isthe same as those of the first and second embodiments, and thereforeredundant description will be omitted, and the difference will bedescribed.

In the second embodiment, an addition average of pixel values of pixelsof the same exposure time and of the same color component that arepresent in the vicinity is calculated in the separating unit 102, and isoutput to the RAW encoding unit 103. In the third embodiment, in orderto further reduce the data amount relative to the second embodiment, again is applied to RAW data of one exposure time in accordance with RAWdata of the other exposure time, and the differences therebetween areoutput to a RAW encoding unit 103. That is, the RAW encoding unit 103encodes, with respect to one exposure time, RAW data of additionaverages, and encodes, with respect to the other exposure time, RAW data(difference RAW data) of difference values.

The processing in the separating unit 102 in the present embodiment willbe described with reference to FIG. 11. First, the separating unit 102adds pixel values of pixels of the same exposure time and of the samecolor component that are present in the vicinity, and obtains RAW data1001 a and 1001 b by calculating the average thereof, similarly to thesecond embodiment, as shown in FIGS. 10A and 10B. Next, differencevalues between a first row in the long exposure RAW data 1001 b andvalues obtained by multiplying a first row of the short exposure RAWdata 1001 a by a gain γ corresponding to the long exposure RAW data andalso adding an offset ε, are obtained. Here, the gain γ and the offset εmay be determined by performing calculation backwardly from the exposuretimes in advance, or may be determined using a histogram of obtainedpixel values of the short exposure pixels and the long exposure pixel.

Specifically, the difference in the first row is calculated as shown inthe following Formulas 23 to 26.

R″ (1,1)−{γ R′ (1,1)+ε}=ΔR(1,1)  Formula 23

G1″(2,1)−{γ G1′(2,1)+ε}=ΔG1(2,1)  Formula 24

G2″(3,1)−{γ G2′(3,1)+ε}=ΔG2(3,1)  Formula 25

B″ (4,1)−{γ B′ (4,1)+ε}=ΔB(4,1)  Formula 26

This operation is similarly performed with respect to second, third, andfourth rows, in addition to the first row, and the calculated differencevalues are output to the RAW encoding unit 103. Note that, in thepresent embodiment, correction is made with respect to the shortexposure RAW data, but correction may be made with respect to the longexposure RAW data. However, from the viewpoint of rounding processing,the accuracy of the difference is better when the gain y is applied tothe short exposure RAW data.

As described above, in the third embodiment, instead of outputting RAWdata as is to the RAW encoding unit 103, the RAW data is output asdifference values, and as a result, the recording data amount of RAWdata can further be reduced relative to the second embodiment.

Fourth Embodiment

Next, a fourth embodiment of the disclosure will be described. In thefourth embodiment, a pixel array that is different from those of thefirst to third embodiments, that is, specifically, the pixel array shownin FIG. 12 is applied to an image capture unit 101.

In the first to third embodiments described above, the image captureunit 101 having a structure in which a minimum unit includes 4×4 16pixels constituted by four different pixels of R, G1, G2, and B, and theminimum unit is repeatedly arranged, as shown in FIG. 2, has beendescribed.

In contrast, FIG. 12 shows a pixel array and the setting of the exposuretime of the image capture unit 101 in the fourth embodiment. Thehorizontal direction is denoted by x, the vertical direction is denotedby y, the column number is represented by an x coordinate, and the rownumber is represented by a y coordinate. The numbers with parenthesesindicate the coordinates indicating the position of each pixel on theimage sensor. Also, white pixels represent short exposure pixels, andgray pixels represent long exposure pixels. In this way, in FIG. 12, ina pixel array of a Bayer arrangement constituted by an array of R, G1,G2, and, B pixels, short exposure pixels and long exposure pixels arealternatingly set in units of two columns.

In the pixel array and the exposure time setting in FIG. 12 as well, asa result of performing processing while performing rearrangement to thepixel arrangement structure shown in FIG. 2, as shown in FIG. 13, theprocessing described in the first to third embodiments can be performed.

As described above, in the fourth embodiment, processing similar to theprocessing described in the first to third embodiments can be performedeven if the pixel array is changed.

Fifth Embodiment

Next, the detailed configuration of a RAW data encoding unit 103 thatperforms encoding processing of short exposure RAW data 401 a and 401 band long exposure RAW data 401 c and 401 d and the processing flow inthe fifth embodiment will be described with reference to the blockdiagram shown in FIG. 14. Note that the configurations shown in FIGS. 1to 5 are similar to those of the first embodiment.

A RAW data encoding unit 103 mainly includes a channel transform unit1601, a frequency transform unit 1602, a quantization parametergenerating unit 1603, a quantization unit 1604, and an encoding unit1605.

The channel transform unit 1601 transforms RAW data configured as in theBayer arrangement that is input from a separating unit 102 into aplurality of channels. Here, transformation is performed into fourchannels separately for R, G1, G2, and B of the Bayer arrangement.

The frequency transform unit 1602 performs frequency transformprocessing by discrete wavelet transform at a predetermined resolutionlevel (hereinafter, denoted as “lev”) for each channel, and outputsgenerated subband data (transform coefficient) to the quantizationparameter generating unit 1603 and the quantization unit 1604.

FIG. 15A shows a filter bank configuration for realizing the discretewavelet transform regarding the subband division processing at lev=1.When the discrete wavelet transform processing is executed in thehorizontal and vertical directions, division into one low frequencysubband (LL) and three high frequency subbands (HL, LH, HH) isperformed, as shown in FIG. 15B.

The transfer functions of the low pass filter (hereinafter, denoted as“lpf”) and the high pass filter (hereinafter, denoted as “hpf”) shown inFIG. 15A are respectively shown in Formulas 27 and 28.

lpf(Z)=(−Z ⁻²+2Z ⁻¹+6+2Z ¹ −Z ²)/8  Formula 27

hpf(Z)=(−Z ⁻¹+2−Z ¹)/2  Formula 28

When lev is larger than 1, subband division is hierarchically executedwith respect to the low frequency subband (LL). Note that, here, thediscrete wavelet transform is configured by a five tap lpf and a threetap hpf, as shown in Formulas 27 and 28, but there is no limitationthereto, and a filter configuration in which the number of taps and thecoefficients are different may be adopted.

The quantization parameter generating unit 1603 calculates, with respectto subband data (transform coefficient) generated by the frequencytransform unit 1602, a brightness feature amount in units ofpredetermined coefficients (square block of one coefficient or more,square area of one pixel or more), and generates a quantizationparameter according to the feature amount. Quantization is similarlyperformed in units of predetermined coefficients (square block of onecoefficient or more), but it may be desirable to be the same as the unitof calculating the feature amount, considering the controllability ofthe image quality. The method of setting the quantization parameteraccording to the brightness and the flow of generating the quantizationparameter will be described later in detail. Then, the generatedquantization parameters are output to the quantization unit 1604.

The quantization unit 1604 performs quantization processing on thesubband data (transform coefficient) input from the frequency transformunit 1602 using the quantization parameters supplied from thequantization parameter generating unit 1603, and outputs the quantizedsubband data (transform coefficient) to the encoding unit 1605.

The encoding unit 1605 performs predictive difference entropy coding ofthe quantized subband data (transform coefficient) input from thequantization unit 1604 for each subband in a raster scan order, andstores the generated encoded RAW data to a memory 107. Note that othermethods may be used as the prediction method and the entropy codingmethod.

Here, the HDR (high dynamic range) synthesizing processing method willbe described using FIG. 16. FIG. 9 is a processing block diagram forperforming HDR synthesizing. A digital camera 100 is configured so as torecord two sheets of RAW data that are different in exposure amount, andtherefore description is given assuming that the HDR synthesizingprocessing in the present embodiment performs HDR synthesizing on twosheets of RAW data. Note that one of the sheets of exposure RAW data isRAW data obtained by capturing performed at correct exposure. The otheris RAW data obtained at an exposure time that causes overexposure orunderexposure, which is auxiliary data for DR expansion.

A developing processing unit 801 performs developing processing on longexposure RAW data. Then, the generated developed long exposure image isoutput to a gain correction unit 803. A developing processing unit 802performs developing processing on short exposure RAW data. Then, thegenerated developed short exposure image is output to a gain correctionunit 804.

A gain correction unit 803 performs gain correction on the long exposureimage using a gain value based on a predetermined synthesizing ratio.The synthesizing ratio will be described later. A gain correction unit804 performs gain correction on the short exposure image using a gainvalue based on the predetermined synthesizing ratio. The synthesizingratio will be described later. An addition processing unit 805 performsaddition processing of pixels at the same coordinate position, withrespect to the long exposure image and the short exposure image.

In this way, in the HDR synthesizing processing, the gain correctionprocessing and the addition processing are performed on images generatedby performing developing processing on two sheets of RAW data that aredifferent in exposure amount. Note that this HDR synthesizing processingis similarly performed on each color component (R, G, B) thatconstitutes image data. Also, the developing processing includesdebayering processing, luminance color difference transform processing,noise removal processing, optical distortion correction processing, andthe like.

Next the synthesizing ratio between the short exposure image data andthe long exposure image data will be described. The way of thinking ofthe synthesizing ratio differs based on which piece of exposure imagedata is image data of correct exposure. The case where long exposureimage data is of correct exposure and the case where short exposureimage data is of correct exposure will be separately described.

First, the synthesizing ratio in the case of the long exposure imagedata being of correct exposure will be described. When the long exposureimage data is obtained by capturing performed at correct exposure, theexposure time of the short exposure image data is relatively shorterthan that of the long exposure image data, and therefore the shortexposure image data is of underexposure.

An example of the histogram of image data when capturing is performed atthis exposure condition is shown in FIG. 17A. The histogram shown inFIG. 17A is a histogram of a specific color component that constitutesthe image data. The horizontal axis of the histogram shows a pixel valueindicating the brightness of image data, and the vertical axis shows thenumber of pixels. Also, Ta and Tb represent pixel threshold values, andTc represents pixel upper limit value. It is defined that an area thatsatisfies the condition of pixel value≤Ta is called as a dark portion,an area that satisfies the condition of Ta<pixel value≤Tb is called asan intermediate portion, and an area that satisfies the condition ofTb<pixel value is called as a bright portion. In this histogram, thelong exposure image data correctly expresses the tone in the darkportion area and the intermediate portion area, but in the brightportion area, many pixels are present in an area at Tc and more, Tcbeing the pixel upper limit, and therefore the long exposure image datais in a state in which tone information is lost due to the occurrence ofblown out highlights. In the HDR synthesizing processing, in order toexpand the tone range in which blown out highlights has occurred, theshort exposure image data at the same coordinate positions issynthesized. In the HDR synthesizing processing in this exposurecondition, addition processing is performed by performing gaincorrection such that the synthesizing ratio of the long exposure imagedata is large in the dark portion area and the intermediate portion areaat which DR can be secured at correct exposure, and the synthesizingratio of the short exposure image data increases in the bright portionarea at which DR is difficult to be secured at correct exposure.

An example of the synthesizing ratio is shown in FIG. 17B. Thehorizontal axis shows the pixel value of the long exposure image data(correct exposure), and the vertical axis shows the synthesizing ratio.The graph in FIG. 17B shows the synthesizing ratios of the pieces ofexposure image data according to the pixel value, and the synthesizingratios of the pieces of exposure image data change such that the sumthereof is constantly 100%. As described in FIG. 17A, because the brightportion includes many pixels at which blown out highlights has occurred,in the graph in FIG. 17B, the synthesizing ratio of the long exposureimage data decreases, from the pixel value at the threshold value Tb, to0% at the pixel upper limit value Tc, and the synthesizing ratio of theshort exposure image data increases, from the pixel value at thethreshold value Tb, to 100% at the pixel upper limit value Tc. As aresult of using such synthesizing ratios, it is possible to expand DR inthe synthesized image while reducing the influence of blown outhighlights. Note that an example has been described in which thesynthesizing ratios change with the threshold value Tb being theboundary in order to make the description easier to understand, but thesynthesizing ratios of the pieces of exposure image data are not limitedthereto.

Based on the above description, the magnitude relationship betweensynthesizing ratios of the long exposure image data and the shortexposure image data is shown in FIG. 17C. A0 in the diagram representsthe synthesizing ratio in a dark portion of the long exposure pixel, A1represent the synthesizing ratio in an intermediate portion of the longexposure pixel, A2 represent the synthesizing ratio in a bright portionof the long exposure pixel. Also, A3 in the diagram represents thesynthesizing ratio in a dark portion of the short exposure pixel, A4represent the synthesizing ratio in an intermediate portion of the shortexposure pixel, A5 represent the synthesizing ratio in a bright portionof the short exposure pixel. The magnitude relationships between thesynthesizing ratios for the respective brightness areas are A0>A3 in thedark portion, A1>A4 in the intermediate portion, and A2<A5 in the brightportion.

Next, the synthesizing ratios in the case of the short exposure imagedata being of correct exposure will be described. When the shortexposure image data is obtained by capturing performed at correctexposure, the exposure time of the long exposure image data isrelatively longer than that of the short exposure image data, andtherefore the long exposure image data is of overexposure.

An example of the histogram of image data when capturing is performed atthis exposure condition is shown in FIG. 18A. The histogram shown inFIG. 18A is a histogram of a specific color component that constitutesthe image data. The horizontal axis of the histogram shows a pixel valueindicating the brightness of image data, and the vertical axis shows thenumber of pixels. Also, Ta and Tb represent pixel threshold values, andTd represents pixel lower limit value. It is defined that an area thatsatisfies the condition of pixel value≤Ta is called as a dark portion,an area that satisfies the condition of Ta<pixel value≤Tb is called asan intermediate portion, and an area that satisfies the condition ofTb<pixel value is called as a bright portion. In this histogram, theshort exposure image data correctly expresses the tone in theintermediate portion area and the bright portion area, but in the darkportion area, many pixels are present in an area at Td and less, Tdbeing the pixel lower limit, and therefore the short exposure image datais in a state in which tone information is lost due to the occurrence ofblocked up shadows. In the HDR synthesizing processing, in order toexpand the tone range at which blocked up shadows has occurred, the longexposure image data at the same coordinate positions is synthesized. Inthe HDR synthesizing processing in this exposure condition, additionprocessing is performed by performing gain correction such that thesynthesizing ratio of the short exposure image data is large in theintermediate portion area and the bright portion area at which DR can besecured at correct exposure, and the synthesizing ratio of the longexposure image data increases in the dark portion area at which DR isdifficult to be secured at correct exposure.

Next, an example of the synthesizing ratio is shown in FIG. 18B. Thehorizontal axis shows the pixel value of the short exposure image data(correct exposure), and the vertical axis shows the synthesizing ratio.The graph in FIG. 18B shows the synthesizing ratios of the pieces ofexposure image data according to the pixel value, and the synthesizingratios of the pieces of exposure image data change such that the sumthereof is constantly 100%. As described in FIG. 18A, because the darkportion includes many pixels at which blocked up shadows has occurred,in the graph in FIG. 18B, the synthesizing ratio of the long exposureimage data is changed so as to be 100% at the pixel lower limit valueTd, and the synthesizing ratio of the short exposure image data ischanged so as to be 0% at the pixel lower limit value Td. As a result ofusing such synthesizing ratios, it is possible to expand DR in thesynthesized image while reducing the influence of blown out highlights.Note that an example has been described in which the synthesizing ratioschange with the threshold value Ta being the boundary in order to makethe description easier to understand, but the synthesizing ratios of thepieces of exposure image data are not limited thereto.

Based on the above description, the magnitude relationship betweensynthesizing ratios of the long exposure image data and the shortexposure image data is shown in FIG. 18C. B0 in the diagram representsthe synthesizing ratio in a dark portion of the short exposure pixel, B1represent the synthesizing ratio in an intermediate portion of the shortexposure pixel, B2 represent the synthesizing ratio in a bright portionof the short exposure pixel. Also, B3 in the diagram represents thesynthesizing ratio in a dark portion of the long exposure pixel, B4represent the synthesizing ratio in an intermediate portion of the longexposure pixel, B5 represent the synthesizing ratio in a bright portionof the long exposure pixel. The magnitude relationships between thesynthesizing ratios for the respective brightness areas are B0<B3 in thedark portion, B1>B4 in the intermediate portion, and B2>B5 in the brightportion.

As described above, in the HDR synthesizing processing, the synthesizingratios of the pieces of exposure image data change in accordance withwhether or not being of correct exposure and the size of pixel value(brightness). The size of synthesizing ratio indicates a degree ofinfluence on the image quality, and in the area in which thesynthesizing ratio is large, the influence on the image quality islarge, and in the area in which the synthesizing ratio is smaller, theinfluence on the image quality is smaller. Therefore, with respect toRAW data to be compression-recorded, the code amount is to be mostsuitably distributed in accordance with the degree of influence on theimage quality based on the synthesizing ratio in the HDR synthesizingprocessing. That is, it is important to set the quantization parameterssuch that the image quality is secured by assigning a larger amount ofcode to an area in which the synthesizing ratio is larger, and the codeamount is reduced regarding an area in which the synthesizing ratio issmall and the influence on the image quality is small.

Next, basic way of thinking in the quantization parameter generationperformed by the quantization parameter generating unit 1603 will bedescribed. As described above, it is assumed that the weighting of thequantization parameter is performed according to the synthesizing ratioobtained by envisioning the HDR synthesizing processing. The way ofthinking of weighting of the quantization parameter according to thebrightness considering the visual property of an image is added thereto.

The RAW data is subjected to adjustment of the luminance level such asgamma correction processing and tone curve correction processing, in thepost processing after development. When a dark portion in which theoriginal luminance level is small is compared with a bright portion inwhich the original luminance level is large, even if adjustment isperformed to the same luminance level, the change ratio of the pixelvalue is larger in the dark portion. If the quantization processing isperformed with the same quantization parameter with respect to the darkportion and the bright portion, the change ratio of the pixel value islarger in the dark portion, and therefore the quantization error due tothe quantization processing is amplified, and the image qualitydegradation becomes apparent. On the other hand, in the bright portionin which the change ratio of the luminance level is small, the changeratio of the pixel value is also small, and as a result, theamplification degree of the quantization error is small, and the imagequality degradation is not apparent. Quantization of the RAW data is tobe performed considering the quantization error amplified by the postprocessing in order to ensure the image quality after the postprocessing. Also, in the dark portion, contrast is small relative tothat in the bright portion, and the signal level of subband data issmall. Therefore, if coarse quantization is performed regarding the darkportion, the subband data after quantization is likely to be 0. Once thecoefficient becomes 0, the signal cannot be restored in the inversequantization process, and apparent image quality degradation occurs.

With these reasons, control is performed such that the quantizationparameter decreases in the dark portion area in which the image qualitydegradation is likely to be apparent, and the quantization parameterincreases in the bright portion area in which the image qualitydegradation is not likely to be apparent. In the present embodiment, aconfiguration will be described in which quantization tables in whichthe quantization parameters for the respective subbands are compiled areprepared in advance, and the quantization table to be referred to isswitched in accordance with the synthesizing ratio and the brightnessfeature amount. These quantization tables are constituted byquantization parameters for respective pieces of subband data accordingto lev. The quantization parameters for each subband are set such thatthe quantization parameter is smaller in the lower subband in whichimage quality degradation is likely to be apparent. If lev=1, themagnitude relationship between the quantization parameters of therespective subbands is 1LL<1HL=1LH<1HH.

Based on the way of thinking of weighting of the quantization parametersin accordance with brightness, exemplary setting of quantization tablesfor pieces of RAW data obtained by capturing at the respective exposuretimes will be described, regarding following three conditionsseparately. Note that, in the present embodiment, an example will bedescribed in which the brightness feature amount is classified intothree feature areas of dark portion, intermediate portion, and brightportion. Note that the definition of the features to be classified issimilar to those in the histograms in FIGS. 17 and 18.

[Exposure Time is the Same Between Short Exposure RAW Data and LongExposure RAW Data]

In this condition, one piece of RAW data is generated by calculating apixel average for each adjacent four pixels of the same color component(refer to FIG. 5). One piece of RAW data is to be quantized, and becausethe HDR synthesizing processing will not be performed, brightnessfeature classification is performed using the RAW data generated bycalculating pixel averages, and quantization is performed using aquantization table according to the classification result. An exemplarysetting of the quantization table is shown in FIG. 19A. Q0 indicates aquantization table for assuring the image quality in the dark portion,Q1 indicates a quantization table for assuring the image quality in theintermediate portion, and Q2 indicates a quantization table for assuringthe image quality in the bright portion. The magnitude relationshipbetween the quantization tables is as follows.

Q0<Q1<Q2

In this way, a quantization table according to the brightness based onthe visual property is set.

[When Exposure Time is Different Between Short Exposure RAW Data andLong Exposure RAW Data, and Short Exposure RAW Data is of CorrectExposure]

In this condition, the image data is separated into short exposure RAWdata and long exposure RAW data (refer to FIGS. 4A to 4D). An exemplarysetting of quantization tables is shown in FIG. 19B. Because blocked upshadows is likely to occur in the short exposure RAW data obtained bycapturing performed at correct exposure, DR in the dark portion isexpanded using the long exposure RAW data obtained by capturingperformed in the condition of overexposure. The quantization tablesindicated by Q1 and Q2 in the diagram are similar to those in FIG. 19A.Here, two quantization tables indicated by Q3 and Q4 are newly added. Q3indicates a table aiming at suppressing generated code amount assumingthat the area is an area in which the synthesizing ratio in the HDRsynthesizing processing is small, and the influence on the image qualityis small. Q4 indicates a quantization table aiming at assigning a largeamount of code in order to expand DR in the dark portion in whichblocked up shadows is likely to occur in the HDR synthesizingprocessing. The magnitude relationship between the quantization tablesare as follows.

Q0≤Q4<Q1<Q2<Q3

or,

Q0<Q4≤Q1<Q2<Q3

The quantization parameter in Q4 is greater than or equal to thequantization parameter in Q0, and is smaller than the quantizationparameter in Q2. In this way, it becomes possible to ensure the imagequality after HDR synthesizing processing by setting a quantizationtable in which the quantization parameter is relatively small withrespect to a dark portion in the long exposure RAW data regarding whichthe synthesizing ratio is large, in addition to quantization tablesaccording to brightness based on the visual property. On the other hand,as a result of setting a quantization table in which the quantizationparameter is large with respect to a dark portion in the short exposureRAW data regarding which the synthesizing ratio is small and to anintermediate portion and a bright portion in the long exposure RAW data,the data amount can be effectively reduced without dropping the imagequality after HDR synthesizing processing.

[When Exposure Time is Different Between Short Exposure RAW Data andLong Exposure RAW Data, and Long Exposure RAW Data is of CorrectExposure]

In this condition as well, the image data is separated into RAW dataconstituted by short exposure pixels and RAW data constituted by longexposure pixels (refer to FIGS. 4A to 4D). An exemplary setting of thequantization table is shown in FIG. 19C. As described above, blown outhighlights is likely to occur in long exposure pixels for capturing atcorrect exposure, and DR in the bright portion is expanded using shortexposure pixels for capturing at underexposure. The quantization tablesindicated by Q0, Q1, and Q3 are similar to those in FIGS. 19A and 19B.Here, a quantization table indicated by Q5 is newly added. Q5 indicatesa quantization table aiming at assigning a large amount of codes forexpanding DR in the bright portion in which blown out highlights islikely to occur in the HDR synthesizing processing. The magnituderelationship between the quantization tables is as follows.

Q0<Q1≤Q5<Q2<Q3

or,

Q0<Q1<Q5≤Q2<Q3

The quantization parameter in Q5 is less than or equal to thequantization parameter in Q2, and is larger than the quantizationparameter in Q0. It becomes possible to ensure the image quality afterHDR synthesizing processing by setting a quantization table in which thequantization parameter is relatively small regarding a bright portion inthe short exposure RAW data in which the synthesizing ratio is large, inaddition to quantization tables according to brightness based on thevisual property, in this way. On the other hand, as a result of settingquantization tables in which the quantization parameter is largeregarding a bright portion in the long exposure RAW in which thesynthesizing ratio is small and a dark portion and an intermediateportion in the short exposure RAW data, the data amount can beeffectively reduced without dropping the image quality after HDRsynthesizing processing.

Next, the quantization processing procedure will be described using theflowcharts shown in FIGS. 20A-20C. In the present embodiment, it isassumed that lev=1 in order to make the description easier tounderstand, and the brightness feature amount is calculated usingsubband data that constitutes RAW data obtained by capturing at anexposure time that is to be correct exposure.

Calculation of the brightness feature amount and the quantizationprocessing are assumed to be performed in units of one coefficient, andthe operation is performed so as to uniquely determine the quantizationtables to be applied to respective pieces of RAW data of differentexposure times, according to the brightness feature amounts for therespective coefficients (refer to FIG. 19 for the details).

In the present embodiment, the operation mode in which capturing isperformed while changing the exposure time for each pixel is called asan HDR mode, and the operation mode in which capturing is performedwithout changing the exposure time is called as a normal mode. Asdescribed above, in the HDR mode, the horizontal size and the verticalsize of RAW data to be recorded is doubled relative to the normal mode(refer to FIGS. 4A to 5), and therefore the amount of data to besubjected to quantization processing differs between the modes.

In step S1201, the controller 108 determines whether or not theoperation mode of the digital camera 100 is the HDR mode. If it isdetermined to be the HDR mode, the processing is advanced to step S1202,and if not, the processing is advanced to step S1219.

In step S1202, the controller 108 determines whether or not the shortexposure RAW data is of correct exposure. If the short exposure RAW datais of correct exposure, the processing is advanced to step S1203, and ifnot, the processing is advanced to step S1211.

In step S1203, the controller 108 calculates the brightness featureamount using the short exposure subband data that is of correctexposure. The size of the coefficient of a 1LL subband of the G1 (green)component is used as the brightness feature amount. It is because the LLsubband is a DC component, and therefore can represent the brightness,and the reason why the G1 component is used is because the human visualproperty is sensitive in the change in G component, and the G1 componentis important visual information.

In step S1204, the controller 108 determines whether or not the area ofinterest is a dark portion based on the magnitude relationship betweenthe brightness feature amount calculated in step S1203 and predeterminedthreshold values. If it is determined to be a dark portion, theprocessing is advanced to step S1205, and if not, the processing isadvanced to step S1206.

In step S1205, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q3, and determines that the quantization table forthe pieces of color component subband data that constitute the longexposure RAW data is Q4,and executes the quantization processing.

In step S1206, the controller 108 determines whether or not the area ofinterest is an intermediate portion based on the magnitude relationshipbetween the brightness feature amount calculated in step S1203 and thepredetermined threshold values. If it is determined to be anintermediate portion, the processing is advanced to step S1207, and ifnot, the processing is advanced to step S1208.

In step S1207, it is determined that the quantization table for thepieces of color component subband data that constitute the shortexposure RAW data is Q1, and the quantization table for the pieces ofcolor component subband data that constitute the long exposure RAW datais Q3, and executes the quantization processing.

In step S1208, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q2, and the quantization table for the pieces ofcolor component subband data that constitute the long exposure RAW datais Q3, and executes the quantization processing.

In step S1209, the controller 108 determines whether or not thequantization processing is completed with respect to all pieces ofsubband data in the image plane. If the quantization processing iscompleted with respect to all pieces of subband data, the processing isended, and if not, the processing is advanced to step S1210.

In step S1210, the controller 108 updates the quantization processingtarget coefficient. The controller 108, upon completing updating of thecoefficient, returns the processing to step S1203.

In step S1211, the controller 108 calculates the brightness featureamount using the long exposure subband data that is of correct exposure.The size of the coefficient of the 1LL subband of the G1 component isused as the brightness feature amount, similarly to step S1203.

In step S1212, the controller 108 determines whether or not the area ofinterest is a dark portion based on the magnitude relationship betweenthe brightness feature amount calculated in step S1211 and predeterminedthreshold values. If it is determined to be a dark portion, theprocessing is advanced to step S1213, and if not, the processing isadvanced to step S1214.

In step S1213, the quantization table for the pieces of color componentsubband data that constitute the short exposure RAW data is determinedto Q3, and the quantization table for the pieces of color componentsubband data that constitute the long exposure RAW data is determined toQ0, and the quantization processing is executed.

In step S1214, the controller 108 determines whether or not the area ofinterest is an intermediate portion based on the magnitude relationshipbetween the brightness feature amount calculated in step S1211 and thepredetermined threshold values. If it is determined to be anintermediate portion, the processing is advanced to step S1215, and ifnot, the processing is advanced to step S1216.

In step S1215, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q3, and the quantization table for the pieces ofcolor component subband data that constitute the long exposure RAW datais Q1, and executes the quantization processing.

In step S1216, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q5, and the quantization table for the pieces ofcolor component subband data that constitute the long exposure RAW datais Q3, and executes the quantization processing.

In step S1217, the controller 108 determines whether or not thequantization processing is completed with respect to all pieces ofsubband data in the image plane. If the quantization processing iscompleted with respect to all pieces of subband data, the processing isended, and if not, the processing is advanced to step S1218.

In step S1218, the controller 108 updates the quantization processingtarget coefficient. The controller 108, upon completing updating of thecoefficient, returns the processing to step S1211.

In step S1219, because the normal mode is determined, the controller 108calculates the brightness feature amount using the subband data obtainedby performing frequency transform on RAW data generated by performingaddition average. The size of the coefficient of a 1LL subband of the G1component that is obtained by performing addition average is used as thebrightness feature amount, similarly to step S1203.

In step S1220, the controller 108 determines whether or not the area ofinterest is a dark portion based on the magnitude relationship betweenthe brightness feature amount calculated in step S1219 and predeterminedthreshold values. If it is determined to be a dark portion, theprocessing is advanced to step S1221, and if not, the processing isadvanced to step S1222.

In step S1221, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the RAWdata is Q0, and executes the quantization processing.

In step S1222, the controller 108 determines whether or not the area ofinterest is an intermediate portion based on the magnitude relationshipbetween the brightness feature amount calculated in step S1219 and thepredetermined threshold values. If it is determined to be anintermediate portion, the processing is advanced to step S1223, and ifnot, the processing is advanced to step S1224.

In step S1223, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the RAWdata is Q1, and executes the quantization processing.

In step S1224, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the RAWdata is Q2, and executes the quantization processing.

In step S1225, the controller 108 determines whether or not thequantization processing is completed with respect to all pieces ofsubband data in the image plane. If the quantization processing iscompleted with respect to all pieces of subband data, the processing isended, and if not, the processing is advanced to step S1226.

In step S1226, the controller 108 updates the quantization processingtarget coefficient. The controller 108, upon completing updating of thecoefficient, returns the processing to step S1219.

As described above, in the present embodiment, the separating unit 102separates RAW data into pieces of data of respective exposure times, thelevel difference between pixels that are different in the exposure timeis eliminated, and with this, a high frequency component is suppressed,and as a result, the recording data amount of RAW data can be reduced.Also, as a result of performing weighting of quantization parametersconsidering the synthesizing ratio while envisioning the HDRsynthesizing processing after developing processing, the recording dataamount of the RAW data can be effectively reduced.

Note that, in the present embodiment, an example in which the brightnessfeature is classified into three stages has been described, but thenumber of stages into which classification is performed is not limitedthereto, and the number of stages may further be increased. Also, in theflowcharts shown in FIGS. 20A-20C, a configuration has been described inwhich, based on the feature amount calculated using the 1LL subband dataof the G1 component, quantization tables for pieces of subband data ofthe other color components are uniquely determined. However, theoperation may be performed such that the quantization table isdetermined by independently calculating the feature amount with respectto each color component.

Also, an example has been described in which the calculation unit of thefeature amount and the processing unit of quantization are each onecoefficient, but the processing unit may be a coefficient block (twocoefficients or more).

Also, an example of lev=1 is described in the flowcharts shown in FIGS.20A-20C, but in the case of lev=2 or more, the horizontal and verticalsizes of subband data differ according to lev. Therefore, thecalculation unit of the feature amount cannot be the same as theprocessing unit of quantization. Assume that the feature amount iscalculated in units of one coefficient of 2LL subband data at lev=2. Inthis case, a 2×2 block is to be set as a processing unit of quantizationwith respect to subband data at lev=1, due to the property ofsubsampling in frequency resolution.

Also, the size of the coefficient of the 1LL subband data is used as thebrightness feature amount, but the feature amount representing thebrightness may be generated using other methods such as using an averagevalue or a pixel that is calculated from coefficients of 1LL subbanddata of a plurality of color components, and there is no limitation tothe method described above.

Also, the channel transform unit 1601 has been described using anexample in which transformation into four channels is performed for eachof color elements of R, G1, G2, and B in the Bayer arrangement, but thecolor elements of R, G1, G2, and B may further be transformed into fourchannels using the following transform formulas 29 to 32.

Y=(R+G1+G2+B)/4  Formula 29

C0=R−B  Formula 30

C1=(G0+G1)/2−(R+B)/2  Formula 31

C2=G0−G1  Formula 32

The above transform formulas illustrate an exemplary transformation intofour channels that are constituted by luminance and color differences.In this case, if control is performed such that the quantizationparameter of the luminance component is reduced, and the quantizationparameters with respect to the other color difference components areincreased, by utilizing the human visual property, the coding efficiencyis improved. Note that the number of channels and the transform methodare not limited thereto.

Sixth Embodiment

Next, a sixth embodiment will be described. In the sixth embodiment, themethod of determining the quantization table for a feature area in whichsynthesizing ratio is large, with respect to RAW data that is not ofcorrect exposure, is different from that of the first embodiment. In thefirst embodiment, a fixed pattern that is prepared in advance is set asthe quantization table for a feature area, of RAW data that is not ofcorrect exposure, in which the synthesizing ratio is large. Therefore,if each exposure RAW data is obtained by capturing performed at anexposure time that is extremely different from that of correct exposure,a most suitable quantization table cannot be selected according to thebrightness, and it is possible that image quality degradation isincurred, or the amount of code is unnecessarily increased. Therefore,in the present embodiment, a method of further increasing the codingefficiency will be described. In the method, with respect to a featurearea in which the synthesizing ratio is large, brightness featuredetermination is also performed on RAW data that is not of correctexposure, and a quantization table that is most suitable according tothe feature is selected. Note that the configuration of an image captureapparatus of the sixth embodiment is similar to the configuration of thefifth embodiment, and therefore the description thereof is omitted.

The quantization processing procedure of the present embodiment is shownin FIGS. 21A-21C. The differences from the fifth embodiment are thatprocessing steps S1301 to S1312 are added. The description of processingsteps that are similar to those of the fifth embodiment is omitted, andonly the differences will be described.

In step S1301, the controller 108 calculates a brightness feature amountusing long exposure subband data that is of overexposure. The size ofthe coefficient of a 1LL subband of the G1 component is used as thebrightness feature amount, similarly to the fifth embodiment.

In step S1302, the controller 108 determines whether or not the area ofinterest is a dark portion based on the magnitude relationship betweenthe brightness feature amount calculated in step S1301 and predeterminedthreshold values. If it is determined to be a dark portion, theprocessing is advanced to step S1303, and if not, the processing isadvanced to step S1304.

In step S1303, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the longexposure RAW data is Q0, and executes the quantization processing.

In step S1304, the controller 108 determines whether or not the area ofinterest is an intermediate portion based on the magnitude relationshipbetween the brightness feature amount calculated in step S1301 and thepredetermined threshold values. If it is determined to be anintermediate portion, the processing is advanced to step S1305, and ifnot, the processing is advanced to step S1306.

In step S1305, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the longexposure RAW data is Q1, and executes the quantization processing.

In step S1306, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the longexposure RAW data is Q2, and executes the quantization processing.

In step S1307, the controller 108 calculates the brightness featureamount using short exposure subband data that is of underexposure. Thesize of the coefficient of a 1LL subband of the G1 component is used asthe brightness feature amount, similarly to the fifth embodiment.

In step S1308, the controller 108 determines whether or not the area ofinterest is a dark portion based on the magnitude relationship betweenthe brightness feature amount calculated in step S1307 and predeterminedthreshold values. If it is determined to be a dark portion, theprocessing is advanced to step S1309, and if not, the processing isadvanced to step S1310.

In step S1309, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q0, and executes the quantization processing.

In step S1310, the controller 108 determines whether or not the area ofinterest is an intermediate portion based on the magnitude relationshipbetween the brightness feature amount calculated in step S1307 and thepredetermined threshold values. If it is determined to be anintermediate portion, the processing is advanced to step S1311, and ifnot, the processing is advanced to step S1312.

In step S1311, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q1, and executes the quantization processing.

In step S1312, the controller 108 determines that the quantization tablefor the pieces of color component subband data that constitute the shortexposure RAW data is Q2, and executes the quantization processing.

As described above, as a result of setting a most suitable quantizationtable according to the brightness with respect to RAW data obtained bycapturing performed at an exposure time that is not of correct exposureas well, the coding efficiency can further be improved.

Other Embodiments

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application Nos.2020-094912, filed May 29, 2020, and No. 2020-188302, filed Nov. 11,2020 which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An apparatus comprising: at least one processor;and a memory coupled to the at least one processor, the memory havinginstructions that, when executed by the at least processor, performoperations as: a generating unit configured to generate a plurality ofpieces of RAW data for respective exposure times from RAW data obtainedfrom a sensor that can perform shooting at an exposure time that isdifferent for each pixel; and an encoding unit configured to encode thegenerated plurality of pieces of RAW data.
 2. The apparatus according toclaim 1, wherein the generating unit generates RAW data in a Bayerarrangement structure for each exposure time.
 3. The apparatus accordingto claim 1, wherein the exposure time that is different for each pixelis constituted by two types, namely a first exposure time and a secondexposure time that is longer than the first exposure time.
 4. Theapparatus according to claim 3, wherein the generating unit generatestwo pieces of RAW data corresponding to the first exposure time and twopieces of RAW data corresponding to the second exposure time, and theencoding unit encodes the two pieces of RAW data corresponding to thefirst exposure time and the two pieces of RAW data corresponding to thesecond exposure time.
 5. The apparatus according to claim 3, wherein thegenerating unit generates one piece of RAW data corresponding to thefirst exposure time by adding a plurality of pieces of pixel data of thefirst exposure time, and generates one piece of RAW data correspondingto the second exposure time by adding a plurality of pieces of pixeldata of the second exposure time, and the encoding unit encodes the onepiece of RAW data corresponding to the first exposure time and the onepiece of RAW data corresponding to the second exposure time.
 6. Theapparatus according to claim 5, wherein the generating unit generatesthe one piece of RAW data corresponding to the first exposure time andthe one piece of RAW data corresponding to the second exposure time bycalculating addition averages of a plurality of pieces of pixel data. 7.The apparatus according to claim 3, wherein the generating unitgenerates difference RAW data from differences between RAW data obtainedby applying a gain to RAW data corresponding to one of the firstexposure time and the second exposure time and RAW data corresponding tothe other of the first exposure time and the second exposure time, andthe encoding unit encodes the difference RAW data and RAW datacorresponding to the other of the first exposure time and the secondexposure time.
 8. The apparatus according to claim 7, wherein thegenerating unit applies a gain to RAW data corresponding to one of thefirst exposure time and the second exposure time so as to be closer toRAW data corresponding to the other of the first exposure time and thesecond exposure time.
 9. The apparatus according to claim 1, wherein thegenerating unit generates the plurality of pieces of RAW data bycalculating addition averages of signals of pixels of a same exposuretime and a same color that are present in the vicinity.
 10. Theapparatus according to claim 1, further perform operation as a controlunit configured to control the exposure time for each pixel of thesensor, wherein the generating unit generates the plurality of pieces ofRAW data if the exposure time for each pixel changes, and generates onepiece of RAW data if the exposure time for each pixel does not change.11. The apparatus according to claim 10, wherein the generating unitgenerates, if the exposure time for each pixel does not change, onepiece of RAW data by calculating average values of image data of pixelsof a same color that are present in the vicinity.
 12. The apparatusaccording to claim 3, wherein the encoding unit determines, using aquantization parameter of RAW data corresponding to one of the firstexposure time and the second exposure time as a reference, aquantization parameter of RAW data corresponding to the other of thefirst exposure time and the second exposure time.
 13. The apparatusaccording to claim 12, wherein the encoding unit determines, using aquantization parameter of RAW data corresponding to the first exposuretime as a reference, a quantization parameter of RAW data correspondingto the second exposure time.
 14. The apparatus according to claim 1,wherein the generating unit generates first RAW data of a first exposuretime and second RAW data of a second exposure time, wherein theinstructions further perform operations as quantization unit configuredto quantize the first RAW data and second RAW data, the encoding unitencodes the first RAW data and second RAW data that have been quantizedby the quantization unit, and the quantization unit determinesquantization parameters for the first RAW data and quantizationparameters for the second RAW data for respective areas that areclassified by brightness of the first RAW data.
 15. The apparatusaccording to claim 14, wherein the quantization unit determines which ofthe first RAW data and the second RAW data is of correct exposure, ifthe first RAW data is of correct exposure, determines quantizationparameters for the first RAW data and quantization parameters for thesecond RAW data for respective areas that are classified by brightnessof the first RAW data, and if the second RAW data is of correctexposure, determines quantization parameters for the first RAW data andquantization parameters for the second RAW data for respective areasthat are classified by brightness of the second RAW data.
 16. Theapparatus according to claim 15, wherein the first exposure time isshorter than the second exposure time.
 17. The apparatus according toclaim 16, wherein the quantization unit determines, with respect to thefirst RAW data, a quantization parameter of an area classified into darkto be a quantization parameter that is larger than a quantizationparameter of an area classified into bright, and determines, withrespect to the second RAW data, a quantization parameter of an areaclassified into bright to be a quantization parameter that is largerthan a quantization parameter of an area classified into dark.
 18. Theapparatus according to claim 15, wherein, when capturing is performed bythe sensor at a same exposure time without performing capturing at anexposure time that is different for each pixel, the generating unitobtains third RAW data obtained by averaging pieces of pixel data ofpixels of a same color that are present in the vicinity, thequantization unit determines quantization parameters for the third RAWdata for respective areas that are classified by brightness of the thirdRAW data, and quantizes the third RAW data, and the encoding unitencodes the quantized third RAW data.
 19. The apparatus according toclaim 18, wherein the quantization unit determines that, if the firstRAW data is of correct exposure, the quantization parameter for thefirst RAW data of an area that is classified as bright is a quantizationparameter corresponding to the quantization parameter that is used foran area classified as bright in the third RAW data, and determines that,if the second RAW data is of correct exposure, the quantizationparameter for the second RAW data of an area that is classified as darkis a quantization parameter corresponding to the quantization parameterthat is used for an area classified as dark in the third RAW data. 20.The apparatus according to claim 19, wherein the quantization unitdetermines that, if the first RAW data is of correct exposure, thequantization parameter for the first RAW data of an area that isclassified as bright is a quantization parameter that is larger than thequantization parameter to be used in the third RAW data, and determinesthat, if the second RAW data is of correct exposure, the quantizationparameter for the second RAW data of an area that is classified as darkis a quantization parameter that is larger than the quantizationparameter to be used in the third RAW data.
 21. The apparatus accordingto claim 18, wherein the quantization unit determines that, if the firstRAW data is of correct exposure, the quantization parameter for thesecond RAW data of an area classified as dark is a quantizationparameter that is greater than or equal to the quantization parameter tobe used for an area that is classified as dark in the third RAW data andis smaller than the quantization parameter to be used for an area thatis classified as bright in the third RAW data.
 22. The apparatusaccording to claim 18, wherein the quantization unit determines that, ifthe second RAW data is of correct exposure, the quantization parameterfor the first RAW data of an area classified as bright is a quantizationparameter that is less than or equal to the quantization parameter to beused for an area that is classified as bright in the third RAW data andis larger than the quantization parameter to be used for an area that isclassified as dark in the third RAW data.
 23. The apparatus according toclaim 14, wherein the quantization unit performs classification bybrightness for each area of the first RAW data or the second RAW datausing a first threshold value for determining whether or not to be adark portion and a second threshold value for determining whether or notto be a bright portion.
 24. The apparatus according to claim 14, whereinthe area is a square area of one pixel or more.
 25. An apparatuscomprising: sensor that can control the exposure time for each pixel;and an encoding apparatus comprising: at least one processor; and amemory coupled to the at least one processor, the memory havinginstructions that, when executed by the at least processor, performoperations as: a generating unit configured to generate a plurality ofpieces of RAW data for respective exposure times from RAW data obtainedfrom a sensor that can perform shooting at an exposure time that isdifferent for each pixel; and an encoding unit configured to encode thegenerated plurality of pieces of RAW data.
 26. A method comprising:generating a plurality of pieces of RAW data for respective exposuretimes from RAW data obtained from a sensor that can perform shooting atan exposure time that is different for each pixel; and encoding thegenerated plurality of pieces of RAW data.
 27. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to execute a method, the method comprising: generating aplurality of pieces of RAW data for respective exposure times from RAWdata obtained from a that can perform shooting at an exposure time thatis different for each pixel; and encoding the generated plurality ofpieces of RAW data.